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Background 

A position paper released on December 5, 2014 by the Philadelphia School Advocacy Partners (PSAP), 
the advocacy arm of the Philadelphia School Partnership (PSP), calls for an “aggressive expansion of 
schools that are achieving results for low-income and minority students” (p. 2). The document describes 
what it terms “high-impact” schools and “underperforming” schools. The author(s) states: 

For poor and minority students in Philadelphia, there really are two kinds of schools: those 
that work and those that don’t... variation in outcomes is not dependent on school type, student 
income levels, or other out-of -school factors (p. 2). 

Research for Action (RFA) found a number of issues with the claims made by PSAP. Specifically: 

1. PSAP argued that the populations served by the two groups of schools are essentially 
identical, but omits discussion of several important differences between the groups— 
primarily in the special education population and in the grade levels served. 

2. PSAP made questionable decisions about which schools to include in their 
“underperforming” sample and failed to detail their rationale for doing so. 

3. We were not able to reconstruct PSAP’s calculations of the percentage of students who are 
eligible for free lunch. 

To be clear, this brief is not intended to offer an alternative explanation for the data presented by PSAP. 
Rather, this document argues that the data presented by PSAP are not nearly sufficient to support their 
sweeping conclusions. 

We begin with an explanation of how PSAP appears to have constructed its sample. 


PSAP’s Sample 


PSAP labels schools “high impact” when their student populations are at least 80% economically 
disadvantaged AND they achieve a state School Performance Profile (SPP) score of 70 or above— the 
Pennsylvania Department of Education’s (PDE’s) cutoff for schools “on the right track.” There are 17 
Philadelphia public schools (both charter and traditional) that meet PSAP’s criteria. 

Meanwhile, schools scoring 40 or below on SPP are classified as “underperforming.” However, PSAP 
does not apply the same criteria for economically disadvantaged (at least 80%) in this instance. It is 
unclear why PSAP uses two different standards in constructing the school samples. Based on PSAP’s 
definition, there are 22 schools in the underperforming system, also including both charter schools and 
traditional public schools. 

In addition, PSAP’s sample includes a significant number of schools that took on the enormous 
challenge of transitioning and serving thousands of students displaced by the district’s 2012-13 school 
building closures. PSAP’s list of “underperformers” includes 11 such receiver schools. No 
schools in the high impact list share this characteristic. 

Methodological Concerns 

For a study to be credible, it must meet a set of quality standards that ensure rigor, consistency, and 
transparency. Our examination of the PSAP document raises questions about whether the analysis 
meets these criteria. 

1. Schools in the PSAP samples do not serve similar students. 

PSAP uses only percent economically disadvantaged in identifying “high-need” students, and even this 
standard is used inconsistently, as noted above. The implicit assumption is that student bodies with 
similar percentages of economically disadvantaged students are also comparable with respect to other 
factors that affect the difficulty and cost of providing an adequate education (e.g., grade level, English 
Language Learner status, special education). There is no basis for this assumption; students differ on a 
wide number of dimensions even within income level. Some of these factors are recorded in state data 
systems and some are not, but PSAP did not include even those factors that are readily available. 

A very brief review of additional student demographics shows a number of disparities between “high- 
impact” and “underperformer” schools, which may, in part, explain the disparity between the two 
groups. Table 1 provides an expanded version of PSAP’s comparison. 
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Table 1 . Underperforming vs. High Impact Schools 



“U nderp erforming” 
schools 

“High-Impact” 

schools 

Panel A. Reported by PSP 

Enrollment (confirmed) 

16,397 

10,088 

Minority (confirmed) 

93-1% 

96.6% 

Economically Disadvantaged 
(confirmed) 

90.8% 

88.3% 

Qualify for Free Lunch, as 
reported by PSP 

76.9% 

76.0% 

Panel B. Not reported by PSP 

Qualify for Free Lunch, our 
calculations 

76.9% 

69.7% 

Qualify for Reduced Lunch 

6.7% 

7 - 5 % 

Qualify for Free or Reduced 
Lunch 

83.6% 

77.2% 

Special Education 

22.5% 

13.2% 

English Language Learners 

7.2% 

6.0% 

Serving High School Students 

18 (of 22) 

4 (of 17) 


As shown in Table l, PSAP did not include percentage enrollment of students in special education 
programs. Twenty-three percent (23%) of students in “underperforming” schools are in special 
education programs, nearly double the rate (13%) at “high-impact” schools. 

This is a significant omission. A more sophisticated approach would have included 
examination of special education rates overall, as well as data on incidence, or severity, 
of student needs. Research by the Education Law Center of Pennsylvania has shown that 
even among special education students, charter schools tend to enroll students whose 
needs are less costly to accommodate . 1 

PSAP also failed to report differences in the percentage of students who are English Language Learners. 
The “underperforming” schools have about 20% more English Language Learner students than the 
“high-impact” schools. PSAP should have accounted for English Language Learner status in 
identifying schools for comparison purposes. 

Further, PSAP’s paper did not differentiate between elementary and secondary schools. Eighty-two 
percent (18/ 22) of schools on the “underperforming” list include students enrolled in high school; the 


1 http://www.elc-pa.org/wp-content/uploads/2014/05/ELC_StudentsDisabilities_Charters_2013.pdf 
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same is true for just 24% (4/17) of schools on the “high-impact” list. Across the state, high schools 
perform far worse on state tests than non-high schools. 

For more valid comparisons, PSAP should have, at minimum, looked at the performance 
of students in the same grades or levels, so that inferences drawn from SPP scores would 
be based on the same set of academic standards and assessments. 

2. PSAP did not explain other inconsistencies in the construction of its two 
samples. 

PSAP included the district’s cyber school, the Virtual Academy, in the “underperforming” schools list, 
while excluding two cyber charters (ACT Academy and ASPIRA Bilingual) that also draw 
overwhelmingly from the district: 97% of enrollment at ASPIRA and 77% of enrollment at ACT originate 
within the district. 2 At minimum, either ASPIRA should have been listed as one of the 
“underperforming” schools, or the district virtual school should have been excluded. The inclusion of 
the two cyber charters would have further increased the average percent economically disadvantaged in 
the “underperforming” group; in contrast, including only the district virtual school lowered it. 

The SPP data contains a clear error in the reported percentage of economically disadvantaged students 
at one of the “underperforming” schools, Universal Audenried Charter School. In 2012-13, the school 
reported that 100 percent of its students were economically disadvantaged, and PDE data shows that 
91% of Audenried students are eligible for free and reduced-price lunch. But in PDEs 2013-14 SPP data, 
the rate of economically disadvantaged students is reported as 23%. The reason for the discrepancy is 
unclear, but using the 2013-14 SPP figure dramatically deflates the percentage of economically 
disadvantaged students in the “underperforming” sample. 

PSAP included only schools with 80% or more economically disadvantaged students in the “high- 
impact” group, but did not apply this rule for the “underperforming” group. In doing so, PSAP included 
several schools with relatively low percentages of economically disadvantaged students in its group of 
’’underperforming” schools, thereby driving down the average percentage of economically 
disadvantaged students in this group. This decision gives the impression that all schools in the 
“underperforming” group are serving essentially the same group of students as the schools in the “high- 
impact” group. However, if the 80% rule is applied consistently to both the ”high-impact” and 
“underperforming” groups of schools, Audenreid and the Virtual Academy (along with three other 
lower-poverty schools) would have been excluded. The remaining schools serve an average 98.8% 
economically disadvantaged students. 

3. We could not replicate PSAP’s calculation regarding free lunch eligibility. 

We were unable to verify PSP’s calculations of the percentage of students receiving free lunch at the 
“high-impact” schools. PSAP reported that 76% of students at the “high-impact” schools are eligible for 
free lunch; our calculations showed that 69.7% were eligible. We calculated our figures using the data 


2 Author calculations based on Department of Education data from 

http://www.portal.state.pa.us/portal/server.pt/community/annual_reports_and_enrollment_data/7357 
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from all schools that reported free and reduced-price lunch figures in the document cited by PSAP. 
Some schools were missing data in both the “high-impact” and “underperforming” schools, but we were 
only able to recreate PSAP’s figures for the latter. 
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